Dynamic Graph Representation Learning for Video Dialog via Multi-Modal Shuffled Transformers

Given an input video, its associated audio, and a brief caption, the audio-visual scene aware dialog (AVSD) task requires agent to indulge in question-answer with human about content. This thus poses challenging multi-modal representation learning reasoning scenario, advancements into which could influence several human-machine interaction applications. To solve this task, we introduce semantics-controlled shuffled Transformer framework, consisting of sequence modules, each taking modality as producing representations conditioned on question. Our proposed variant uses shuffling scheme their multi-head outputs, demonstrating better regularization. encode fine-grained visual information, present novel dynamic graph pipeline that consists intra-frame layer spatio-semantic for every frame, inter-frame aggregation module capturing temporal cues. entire is trained end-to-end. We experiments benchmark AVSD dataset, both answer generation selection tasks. results demonstrate state-of-the-art performances all evaluation metrics.